Image group title assigning device, image grouping device, representative image determination device for image group, image display device, camera, and image display program

ABSTRACT

An image group title assigning device includes: a title candidate assigning means for assigning a plurality of title candidates to an image group having a plurality of images; a characteristic amount calculating means for individually calculating characteristic amounts of a plurality of images included in the image group in association with each of the plurality of title candidates; and a title determining means for determining a title representing the image group from among the title candidates based upon a characteristic amount of each of the images calculated for each of the title candidates.

This is a Continuation Application of application Ser. No. 14/663,335,filed Mar. 19, 2015, which is a Divisional Application of applicationSer. No. 12/811,760 filed Jul. 6, 2010 which is a National Phase ofPCT/JP2008/071867 filed Dec. 2, 2008 which claims priority to JP2008-113709 filed Apr. 24, 2008, JP 2008-113708 filed Apr. 24, 2008, JP2008-113707 filed Apr. 24, 2008, JP 2008-008990 filed Jan. 18, 2008 andJP 2008-113706 Filed Apr. 24, 2008. The disclosures of the priorapplications are hereby incorporate by reference herein in theirentireties.

TECHNICAL FIELD

The present invention relates to a title assigning device, an imagegrouping device, a representative image determination device for animage group, an image display device, a camera, and an image displayprogram.

BACKGROUND ART

(1) There is a known art for distinguishing a semantic classification(for example, a shooting scene) of individual images constituting animage group based upon a characteristic amount of each of the images andfor determining the semantic classification of the said image groupbased upon the characteristic amount of the image group usingdistinguish results with respect to each of the images (refer to patentreference literature 1).

(2) There is a known art for extracting identification information froman image itself or attribute information of the image, and foridentifying an event of the said image based upon the extractedidentification information (refer to patent reference literature 2).

(3) There is a known art for grouping a plurality of images based uponindividual shooting date and time information of the said plurality ofimages. For instance, processing is performed so as to put the imagesinto successive groups in time series based upon shooting frequencyinformation aggregated for each predetermined shooting period (refer topatent reference literature 3).

(4) There is a known art for selecting at image representingcharacteristics of a group from among grouped images (refer to patentreference literature 4). Evaluation values representing the likelihoodof scenery, still life, portrait, or the like are individuallycalculated from the images constituting the group. In the event that thenumber of scenery images is the largest among the images in the group,an image with the largest evaluation value about scenery is selected asthe representative image among the scenery images.

(5) There is a known album creating apparatus as follows. The albumcreating apparatus selects an image of a subject appearing mostfrequently in grouped images as a representative image (main image)(refer to patent reference literature 5).

[Patent reference literature 1] U.S. Pat. No. 7,035,467 [Patentreference literature 2] Japanese Laid Open Patent Publication No.2007-129434 [Patent reference literature 3] Japan Patent No. 3914747[Patent reference literature 4] Japanese Laid Open Patent PublicationNo. 2007-94990 [Patent reference literature 5] Japanese Laid Open PatentPublication No. 2006-259932

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention

(1) Both in processing of individual images and in processing of animage group including these images, the conventional technology uses theSVM (Support Vector Machine) method, which performs distinguishprocessing based upon a characteristic amount thereof. There is an issueyet to be addressed effectively in that load of processing increaseswhen the SVM method is used for an image group including a plurality ofimages.

(2) According to the conventional technology, there is an issue yet tobe addressed effectively in that, as the number of types of titles (anevent name, subject name, or the like) of images to be identifiedincreases, identification information to be extracted increases and loadof identification processing performed based upon the saididentification information increases.

(3) According to the conventional technology, there is an issue yet tobe addressed effectively in that, if a plurality of groups put togetherbased upon shooting frequency information are included in apredetermined shooting period, the said plurality of groups are put intothe same group even if the shooting scene (shooting target) varies amongthe groups.

(4) In the conventional technology, there is an issue yet to beaddressed effectively in that load of calculation processing necessaryto determine a representative image increases with an increase in thenumber of types of evaluation values to be calculated such as scenery,portrait, still life, event, and commemorative photograph.

(5) The conventional devices are not configured in consideration of amethod to select an appropriate image, in the event that there are aplurality of images of a subject appearing most frequently in a group,as a representative image from among them.

Means for Solving the Problems

(1) According to a first aspect of the present invention, an image grouptitle assigning device, comprises: a title candidate assigning means forassigning a plurality of title candidates to an image group having aplurality of images; a characteristic amount calculating means forindividually calculating characteristic amounts of a plurality of imagesincluded in the image group in association with each of the plurality oftitle candidates; and a title determining means for determining a titlerepresenting the image group from among the title candidates based upona characteristic amount of each of the images calculated for each of thetitle candidates.

According to a second aspect of the present invention, in the imagegroup title assigning device according to the first aspect, it is alsopossible that the title determining means calculates, for each of thetitle candidates, probability information of each of the imagescorresponding to the individual title candidates, further calculates asum of probability information of equal to or greater than apredetermined value among the probability information for each of thetitle candidates, and determines a title candidate having a maximumvalue of the sum of the probability information as a title representingthe image group.

According to a third aspect of the present invention, in the image grouptitle assigning device according to the second aspect, if with respectto the title candidate, a number of images having probabilityinformation of equal to or greater than the predetermined value to anumber of images included in the image group falls below a predeterminedproportion, the title determining means may exclude said titlecandidate.

According to a fourth aspect of the present invention, in the imagegroup title assigning, device according to the first aspect, the imagegroup may be grouped according to shooting date and time.

According to a fifth aspect of the present invention, in the image grouptitle assigning device according to the first aspect, the image groupmay be grouped according to shooting position.

According to a sixth aspect of the present invention, in the image grouptitle assigning device according to the fourth or fifth aspect, thetitle candidate assigning means may assign the plurality of titlecandidates in accordance with the shooting date and time or the shootingposition.

According to a seventh aspect of the present invention, a cameracomprises an image group title assigning device according to any one ofthe first to sixth aspects.

(2) According a eighth aspect of the present invention, an image titleassigning device comprises: a title candidate assigning means tinassigning as predetermined number of title candidates to an image; acharacteristic amount calculating means for calculating a characteristicamount of the image in association with each of the predetermined numberof title candidates; and a title determining means for determining atitle of the image from among the title candidates based upon thecharacteristic amount of the image calculated for each of the titlecandidates.

According to a ninth aspect of the present invention, in the image titleassigning device according to the eighth aspect, the title candidateassigning means may select a predetermined number of title candidates inaccordance with shooting date and time information.

According to a tenth aspect of the present invention, in the image titleassigning device according to the eighth or ninth aspect, the titlecandidate assigning means may select a predetermined number of titlecandidates in accordance with shooting position information.

According to a 11th aspect of the present invention, the image titleassigning device according to any one of the eighth to tenth aspects mayfurther comprise a face detection means for detecting a face of a personin the image based upon data of said image. In this case, the titlecandidate assigning means may select a predetermined number of titlecandidates according to whether or not a face has been detected by theface detection means.

According to a 12th aspect of the present invention, the image titleassigning device according to the 11th aspect may further comprise aface identification means for identifying a face of the person. In thiscase, the title candidate assigning means may select a predeterminednumber of title candidates in accordance with an identification resultof a face by the face identification means.

According to a 13th aspect of the present invention, in the image titleassigning device according to any one of the eighth to 12th aspects, itis also possible that the title candidate assigning means assigns apredetermined number of title candidates to an image group having aplurality of images; the characteristic amount calculating meansindividually calculates characteristic amounts of a plurality of imagesincluded in the image group in association with each of thepredetermined numbers of title candidates; and the title determiningmeans determines a title representing the image group from among thetitle candidates based upon the characteristic amount of the imagecalculated for each of the title candidates.

According to a 14th aspect of the present invention, a camera comprisesan image title assigning device according to any one of the eighth to13th aspects.

(3) According to a 15th aspect of the present invention, an imagegrouping device comprises: a characteristic amount calculating means forindividually calculating, for each of image groups having a plurality ofimages, characteristic amounts of a plurality of images included in eachimage group; a title determining means for determining a titlerepresenting each of the image groups based upon characteristic amountsof the individual images; a calculation means for calculating timedifference information between the image groups based upon informationof shooting date and time of individual images included in the pluralityof image groups; and an integration means for putting the image groupsinto one image group if titles determined by the title determining meansare same and time difference information calculated by the calculationmeans is equal to or less than a predetermined value.

According to a 16th aspect of the present invention, in the imagegrouping device according to the 15th aspect, it is also possible thatthe plurality of image groups are each grouped based upon theinformation of shooting date and time; and the characteristic amountcalculating means individually calculates characteristic amountsassociated with the information of shooting date and time.

According to a 17th aspect of the present invention, in the imagegrouping device according to the 15th or 16th aspect, the timedifference information may be a difference between a last shooting timeof an image included in one image group and an earliest shooting time ofan image included in another image group.

According to a 18th aspect of the present invention, in the imagegrouping device comprises: a characteristic amount calculating means forindividually calculating, for each of image groups having a plurality ofimages, characteristic amounts of a plurality of images included in eachimage group: a title determining means for determining a titlerepresenting each of the image groups based upon characteristic amountsof the individual images; a calculation means for calculating distanceinformation between the image groups based upon shooting positioninformation of individual images included in the plurality of imagegroups; and an integration means for putting the image groups into oneimage group if titles determined by the title determining means are sameand distance information calculated by the calculation means is equal toor less than a predetermined value.

According to a 19th aspect of the present invention, in the imagegrouping device according to the 18th aspect, it is also possible thatthe plurality of image groups are each grouped based upon the shootingposition information; and the characteristic amount calculating meansindividually calculates characteristic amounts associated with theshooting position information.

According to a 20th aspect of the present invention, in the imagegrouping device according to the 18th or 19th aspect, the distanceinformation may be a shortest distance between a shooting position of animage included in one image group and a shooting position of an imageincluded in another image group.

According to a 21st aspect of the present invention, a camera comprisesan image grouping device according to any one of the 15th to 20thaspects.

(4) According to a 22nd aspect of the present invention, arepresentative image determination device for image group comprises: adetermination means for determining an image representing an image grouphaving a plurality of images from said image group; and a control meansfor controlling the determination means so as to determine therepresentative image based upon selection criterion information, whichis set according to a title assigned to the image group, to select therepresenting image.

According to a 23rd aspect of the present invention, the representativeimage determination device for image group according to the 22nd aspectmay further comprise a storage means for storing in advance a pluralityof selection criterion information each corresponding to a plurality oftitles. In this case, the control means may read out selection criterioninformation corresponding to a title assigned to the image group fromthe storage means and, based upon said selection criterion information,controls the determination means so as to determine a representativeimage.

According to a 24th aspect of the present invention, in therepresentative image determination device for image group according tothe 23rd aspect, the selection criterion information may includeinformation to instruct processing performed in accordance with thetitle. In this case, the determination means may determine therepresentative image based upon a result of the processing.

According to a 25th aspect of the present invention, in therepresentative image determination device for image group according tothe 24th aspect, the selection criterion information may includeposition information related to an area name included in the title. Inthis case, the determination means may determine the representativeimage based upon shooting position information of individual imagesconstituting the image group and position information read out from thestorage means.

According to a 26th aspect of the present invention, in therepresentative image determination device for image group according tothe 24th aspect, the selection criterion information may includeinformation of color corresponding to the title. In this case, thedetermination means may determine the representative image based uponregion information of the color in individual images constituting theimage group.

According to a 27th aspect of the present invention, in therepresentative image determination device for image group according tothe 24th aspect, processing performed in accordance with the title mayinclude an instruction to obtain subject tracking information related toindividual images constituting the image group. In this case, thedetermination means may determine the representative image based uponthe obtained subject tracking information.

According to a 28th aspect of the present invention, a camera comprisesa representative image determination device according to any one of the22nd to 27th aspects.

(5) According to a 29th aspect of the present invention, an imagedisplay device comprises: a subject recognition means for carrying outsubject recognition processing for image data grouped on a predeterminedcondition in advance and classified and recorded in each group so as torecognize a subject included in an image; a setting means for setting aselection criterion to select a representative image from image datarecorded in each group based upon a recognition result by the subjectrecognition means; and a selection means for selecting therepresentative image from image data recorded in each group based uponthe selection criterion set by the setting means.

According to a 30th aspect of the present invention, it may be arrangedto display information related to the representative image selected bythe selection means and information related to a group that includes therepresentative image in association with each other.

According to a 31st aspect of the present invention, it may be possibleto, as a result of a subject recognition, to make a decision as towhether or not an image in which a person is captured is included ineach group, and, when a decision is made that an image in which a personis captured is included, to set a first selection criterion as theselection criterion, whilst, when a decision is made that an image inwhich a person is captured is not included, to set a second selectioncriterion as the selection criterion.

According to a 32nd aspect of the present invention, the first selectioncriterion may be a selection criterion to select the representativeimage based upon a face of a person captured in an image.

According to a 33rd aspect of the present invention, in the 32nd aspect,it is preferable that the first selection criterion is a selectioncriterion to select the representative image by making a decision as towhether or not a face of the person is a face of a family member or anacquaintance, whether or not a face of the person is facing a front,whether or not a face of the person is a smile, and whether or not anarea of a face of the person is a greatest.

According to a 34th aspect of the present invention, it is preferablethat the second selection criterion is a selection criterion to selectthe representative image by making a decision as to whether or not theimage is not blurring, whether or not the image has a main subject infocus, whether or not the image is with proper brightness, and whetheror not the image is captured in an optimal composition.

Advantageous Effect of the Invention

(1) According to the present invention, load of processing to assign atitle representing an image group can be reduced.

(2) According to the present invention, load of processing to put atitle to an image can be reduced.

(3) According to the present invention, a plurality of images can begrouped appropriately.

(4) According to the present invention, processing necessary todetermine a representative image can be reduced.

(5) According to the present invention, an appropriate image can beselected as a representative image from among images included in agroup.

BRIEF DESCRIPTION OF THE DRAWINGS

[FIG. 1] A block diagram explaining the structure of a main section ofan electronic camera according to an embodiment of the presentinvention.

[FIG. 2] A flowchart explaining the flow of grouping processing executedby a main CPU.

[FIG. 3] A flowchart explaining clustering processing.

[FIG. 4] A flowchart explaining event decision processing for eachimage.

[FIG. 5] A flowchart explaining event decision processing for eachcluster.

[FIG. 6] A flowchart explaining cluster integration processing.

[FIG. 7] A flowchart explaining processing to determine a representativeimage.

[FIG. 8] A figure showing an example of an event candidate table.

[FIG. 9] A figure showing an example of an event decision (field day)for clusters.

[FIG. 10] A figure showing an example of an event decision (wedding) forclusters.

[FIG. 1F] A figure showing an example of an event decision (cherryblossom viewing) for clusters.

[FIG. 12] A figure showing an example of a selection criterioninformation table.

[FIG. 13] A figure showing an example of a computer device.

[FIG. 14] A figure showing an example of an event candidate tableaccording to the second embodiment.

[FIG. 15] A figure showing an example of a selection criterioninformation table according to the third embodiment.

[FIG. 16] A block diagram showing the structure of the image displaydevice according to the fourth embodiment.

[FIG. 17] A schematic illustration of an example of a folder structure.

[FIG. 18] An illustration of an example of a folder structure usingrepresentative images.

[FIG. 19] A flowchart showing processing of the image display device.

[FIG. 20] A flowchart showing the flow of “representative imageselection processing in the event that a person is photographed”.

[FIG. 21] A flowchart showing the flow of “representative imageselection processing in the event that a person is not photographed”.

BEST MODE FOR CARRYING OUT THE INVENTION

The best mode for carrying out the present invention will now beexplained.

First Embodiment

FIG. 1 is a block diagram explaining the structure of the main sectionof an electronic camera 1 according to an embodiment of the presentinvention. The electronic camera 1 is controlled by a main CPU 11.

A photographic lens 21 forms a subject image on an imaging plane of animage sensor 22. The image sensor 22, being constituted with a CCD imagesensor and the like, captures a subject image on the imaging plane andoutputs an image-capturing signal to an imaging circuit 23. Colorfilters of R (red), G (green), and B (blue) are provided on the imagingplane of the image sensor 22 so that each of the color filterscorresponds to a pixel location. Since the image sensor 22 captures thesubject image through the color filters, a photoelectric conversionsignal output from the image sensor 22 includes RGB color information.

The imaging circuit 23 performs analog processing (gain control and thelike) on the photoelectric conversion signal output from the imagesensor 22 and converts an analog image-capturing signal into digitaldata through a built-in A/D conversion circuit.

The main CPU 11 inputs a signal output from each block, carries out apredetermined operation, and outputs a control signal based upon theoperation result to each of the blocks. An image processing circuit 12,being configured, for instance, as an ASIC (application specificintegrated circuit), performs image processing for a digital imagesignal input through the imaging circuit 23. The image processingincludes, for example, edge enhancement, color temperature adjustment(white balance adjustment) processing, and format conversion processingfor the image signal.

An image compression circuit 13 performs image compression processing,for instance, at a predetermined compression ratio in JPEG format on theimage signal having undergone the processing performed by the imageprocessing circuit 12. A display image forming circuit 15 forms displaydata so as to display a photographed image on an LCD monitor 16.

A buffer memory 14 is used so as to temporarily store data before andafter image processing and during image processing, as well as storingan image file before recorded in a recording medium 30 and storing animage file read out from the recording medium 30.

The recording medium 30 is constituted with, for instance, a memory cardthat can be attached to and detached from the electronic camera 1. Inresponse to an instruction from the main CPU 11, data of a capturedimage and an image file containing information of the captured image arerecorded in the recording medium 30. The image file recorded in therecording medium 30 can be read out in response to an instruction fromthe main CPU 11.

A program to be executed by the main CPU 11, data necessary forprocessing performed by the main CPU 11, and the like are stored in aflash memory 19. It is arranged that the contents of the program and thedata stored in the flash memory 19 can be added and modified in responseto an instruction from the main CPU 11.

Operation members 17, including a variety of buttons and switches of thes electronic camera 1, output operation signals to the main CPU 11 inresponse to the operation content of each of the operation members suchas a depression operation of a release button and a switch operation ofa mode changeover switch.

In response to an instruction from the main CPU 11, a GPS device 18receives radio waves from a GPS satellite and outputs a received signalto the main CPU 11. Based upon the received signal from the GPS device18, the main CPU 11 carries out a predetermined operation so as todetect positioning information (latitude, longitude, and altitude) ofthe electronic camera 1.

The electronic camera 1 is configured to carry out predetermined imageprocessing and compression processing for an image signal obtained atthe image sensor 22 when photography is performed and to generate animage file in which additional information including positioninginformation and information related to the said captured image is addedto the image data having undergone the compression processing. Morespecifically, the electronic camera 1 stores image data in JPEG formatin an image data unit and generates an image file in Exif format inwhich the additional information is stored in an additional informationunit. The image file in Exif format is an image file in which athumbnail image and additional information data are embedded in imagedata in JPEG image format. The generated image file is stored in therecording medium 30.

It is arranged that the electronic camera 1 can be switched in between ashooting mode and a reproduction mode. The shooting mode is an operationmode to capture a subject image and save data of the captured image inthe recording medium 30 as an image file. The reproduction mode is amode to display a reproduction image based upon the image data on theLCD monitor 16 by reading out the captured image data from the recordingmedium 30 or the like.

<Grouping of Captured Images>

The electronic camera 1 of the present embodiment includes a function toautomatically group captured images. More specifically, the electroniccamera 1 groups image files recorded in the recording medium 30 andstores the image files into a folder provided for each group. Inaddition, based upon the image group included in each folder, theelectronic camera 1 determines a title (for instance, captured scene)representing the shooting target of the image group. Then, based uponthe title, the electronic camera 1 selects, for each folder (group),individual image files representing an image group of each folder(group). It is to be noted that it may also be arranged that theelectronic camera 1 creates a control table of the image files and,stores grouping information for the image files in the said table, inplace of storing the image files in the folder provided for each group.

FIG. 2 is a flowchart explaining the flow of the grouping processingexecuted by the main CPU 11. Upon input of an operation signalinstructing execution of grouping processing from the operation members17, the main CPU 11 starts processing of FIG. 2.

In a step S10 of FIG. 2, the main CPU 11 uses a clustering method so asto group image files recorded in the recording medium 30 and causes theflow of control to proceed to a step S20. The clustering processing willbe described later in detail. In the step S20, the main CPU 11 performsan event decision for each image included in a cluster (set of groupedimage files) and causes the flow of control to proceed to a step S30.Here, an “Event” represents the shooting scene of an image, including,for instance, “Field Day”, “Wedding”, “Cherry Blossom Viewing”, “SeaBathing”, and “Trip to Hokkaido”. The event decision processing for eachimage will be described later in detail.

In the step S30, the main CPU 11 performs an event decision for eachcluster and causes the flow of control to proceed to a step S40.Although the steps S20 and S30 are the same in that “Event” is a title,they are different in that the event decision targets individual imagesin the step S20 whilst the event decision targets an “Event”representing each cluster in the step S30. The event decision processingfor each cluster will be described later in detail.

In the step S40, the main CPU 11 integrates the clusters according torequirements and causes the flow of control to proceed to a step S50.The cluster integration processing will be described later in detail. Inthe step S50, the main CPU 11 determines an image file representing thecluster and terminates the flow of control of FIG. 2.

<Clustering Processing>

The clustering processing (S10) will now be explained in detail withreference to the flowchart shown as an example in FIG. 3. The clusteringprocessing is performed using, for instance, shooting date and timeinformation. In a step S11 of FIG. 3, the main CPU 11 extracts, for allthe image files, information showing shooting date and time recorded inan additional information unit of the image file and causes the flow ofcontrol to proceed to a step S12.

In the step S12, the main CPU 11 makes a decision as to whether or notinformation showing shooting date and time has been extracted from allthe image files. In the event that necessary information has beenextracted from all the image files, the main CPU 11 makes a positivedecision in the step S12 and causes the flow of control to proceed to astep S13, whilst in the event that necessary information has not beenextracted from all the image files, the main CPU 11 makes a negativedecision in the step S12 and causes the flow of control to return to thestep S11. In the event of returning to the step S11, the extractionprocessing is repeated.

In the step S13, the main CPU 11 uses a hierarchical clustering, inwhich, for instance, the nearest neighbor method is used for clustering.More specifically, with one cluster as a starting point of processingfor each image, the main CPU 11 groups image files into a plurality ofclusters (sets of image files with the similar period of shooting time)by repeating processing to sequentially integrate clusters with thesimilar period of shooting time. In the event that the number ofclusters is reduced to a predetermined number, the main CPU 11 causesthe flow of control to proceed to a step S14. Alternatively, in theevent that the difference between the last shooting time in one set andthe earliest shooting time in the other set exceeds a predeterminedperiod of time (for instance, three hours) in adjacent clusters, themain CPU 11 causes the flow of control to proceed to the step S14. It isto be noted that a method other than the nearest neighbor method may beused among the hierarchical clustering. In addition, a method other thanthose of the hierarchical clustering (for example, apartitioning-optimization method) may also be used.

In the step S14, the main CPU 11 creates folders in the recording medium30 corresponding to the clusters and causes the flow of control toproceed to a step S15. In the step S15, the main CPU 11 movescorresponding image files to the created folders and terminates the flowof control of FIG. 3. As a result, the image files belonging to each ofthe clusters are stored in each of the folders corresponding to theclusters. It is to be noted that it may also be arranged that the mainCPU 11 creates a control table of the image files and stores groupinginformation for the image files in the said table, in place of storingthe image files in the folder provided for each group.

<Event Decision Processing for Each Image>

The event decision processing (S20) in terms of image will now beexplained in detail with reference to the flowchart shown as an examplein FIG. 4. In a step S21 of FIG. 4, the main CPU 11 specifies onecluster from among the plurality of clusters and causes the flow ofcontrol to proceed to a step S22. The specifying order is, for instance,a chronological order with respect to shooting data and time (priorityis given to the cluster having an image file of the earliest shootingtime).

In the step S22, the main CPU 11 determines events (referred to as eventcandidates) to be decision targets. With reference to the eventcandidate table shown as an example in FIG. 8, the main CPU 11 selectsevents corresponding to the month of the shooting date of an image fileconstituting the cluster. For example, in the case where an image filewas captured in May, the main CPU 11 renders “Cherry Blossom Viewing”,“Field Day”, and “Wedding” event candidates. The event candidate table,including events that take place frequently in each month, is created inadvance based upon past events which took place in respective month andis recorded in the flash memory 19.

According to FIG. 8, events that are strongly associated with the monthsand seasons in which they take place are included only in correspondingmonths, whilst events that are weakly associated with the months inwhich they take place such as “Wedding” are included in a plurality ofmonths. If the shooting date of an image file constituting the clusterbelongs to a plurality of months, the main CPU 11 selects an eventcandidate corresponding to a month to which more image files belong, asan example. After determining the event to be a decision target asdescribed above, the main CPU 11 causes the flow of control to proceedto a step S23.

In the step S23, the main CPU 11 specifies one image file from among thespecified image files constituting the cluster and causes the flow ofcontrol to proceed to a step S24. In the step S24, based upon image dataincluded in the specified image file, the main CPU 11 calculates thecharacteristic amount of the said image and causes the flow of controlto proceed to a step S25.

The main CPU 11 calculates the characteristic amount of the image thatis appropriate to make a decision as to the event candidate determinedin the step S22. The relation between the event candidate and thecharacteristic amount to be calculated is tabled in advance and recordedin the flash memory 19. The characteristic amount is, for instance,color information, sharpness information, texture information, patterninformation, brightness information, and the like, which are calculatedbased upon pixel data constituting a predetermined region of the image.In addition, the site of the image and information on a color histogrammay be included as the characteristic amount. Since the characteristicamount calculation is a publicly known technique, detailed descriptionsrelated to the characteristic amount calculation will be curtailed inthe present explanation.

In the step S25 to a step S27, the main CPU 11 uses identifierscorresponding to the event candidates determined in the step S22 so asto calculate the probability of being each event. The identifiers arecharacteristic amount information calculated by machine learning usingthe SVM (Support Vector Machine) method based upon a plurality of sampleimage data. For instance, the identifier for “Cherry Blossom Viewing” ischaracteristic amount information calculated based upon a plurality ofsample images of “Cherry Blossom Viewing” and “Non-Cherry BlossomViewing”. The identifier for “Field Day” is characteristic amountinformation calculated based upon a plurality of sample images of “FieldDay” and “Non-Field Day”. The identifier for “Wedding” is characteristicamount information calculated based upon a plurality of sample images of“Wedding” and “Non-Wedding”. In the present embodiment, the identifiercorresponding to each of the plurality of event candidates are createdin advance and recorded in the flash memory 19. In the presentdescription, an example with three event candidates is explained. Themain CPU 11 calculates, for each image specified in the step S23, theprobability P of being an event listed in the event candidates andcauses the flow of control to proceed to a step S28.

The probability P of being an event corresponds to the distance betweenthe boundary that divides a characteristic amount space represented bythe identifier (for instance, the boundary between the “Field Day”region and the “Non-field Day” region) and the characteristic amountcalculated in the step S14. The probability of being “Field Day” is highif a characteristic amount calculated from an image is located at theback of the characteristic amount region corresponding to “Field Day” inthe characteristic amount space represented by the identifier for “FieldDay” and the distance to the characteristic amount region correspondingto “Non-field Day” is long. On the other hand, the probability of being“Field Day” is low if the characteristic amount calculated from theimage is located at the edge of the characteristic amount regioncorresponding to “Field Day” and the distance to the characteristicamount region corresponding to “Non-field Day” is short. The main CPU 11calculates the probability according to the above distance.

In the step S28, the main CPU 11 makes a decision as to whether or notthe processing has been completed for all the image files in thespecified cluster. In the case of having performed the calculation ofthe characteristic amount and that of the probability P of being anevent for all the images in the cluster, the main CPU 11 makes apositive decision in the step S28 and causes the flow of control toproceed to a step S29. In the case of not having performed thecalculation of the characteristic amount and that of the probability Pof being an event for all the images in the cluster, the main CPU 11makes a negative decision in the step S28 and causes the flow of controlto return to the step S23. When the flow of control returns to the stepS23, the main CPU 11 specifies another image file among the specifiedimage files constituting the cluster and causes the flow of control toproceed to the step S24.

In the step S29, the main CPU 11 makes a decision as to whether or notthe processing has been completed for all the clusters. In the case ofhaving performed the calculation of the characteristic amount and thatof the probability P of being an event for all the clusters, the mainCPU 11 makes a positive decision in the step S29 and terminates the flowof control of FIG. 4. In the case of not having performed thecalculation of the characteristic amount and that of the probability Pof being an event for all the clusters, the main CPU 11 makes a negativedecision in the step S29 and causes the flow of control to return to thestep S21. When the flow of control returns to the step S21, the main CPU11 specifies another cluster and causes the flow of control to proceedto the step S22.

<Event Decision Processing for Each Cluster>

The event decision processing (S30) in terms of cluster will now beexplained in detail with reference to the flowchart shown as an examplein FIG. 5. In a step S31 of FIG. 5, the main CPU 11 specifies onecluster from among the plurality of clusters and causes the flow ofcontrol to proceed to a step S32. The specifying order is, for instance,a chronological order with respect to shooting date and time (priorityis given to the cluster having an image file of the earliest shootingtime).

In the step S32, the main CPU 11 specifies one event to be a decisiontarget from among the event candidates, and then causes the flow ofcontrol to proceed to a step S33. In the step S33, the main CPU 11specifies one image file from among the image files constituting thecluster specified in the S31 and causes the flow of control to proceedto a step S34. In the step S34, the main CPU 11 makes a decision as towhether or not the probability P of being an event with respect to theevent specified in the S32 is equal to or greater than a predetermineddecision threshold Pth. If the probability P of being an event is equalto or greater than the decision threshold Pth, the main CPU 11 makes apositive decision in the step S34 and causes the flow of control toproceed to a step S35. If the probability P of being an event is lessthan the decision threshold Pth, the main CPU 11 makes a negativedecision in the step S34 and causes the flow of control to proceed to astep S36.

In the step S35, the main CPU 11 calculates Psum=Psum+P and causes theflow of control to proceed to a step S36. In the step S36, the main CPU11 makes a decision as to whether or not the processing has beencompleted with respect to all the image files in the specified cluster.In the case of having performed the processing for all the images in thecluster, the main CPU 11 makes a positive decision in the step S36 andcauses the flow of control to proceed to a step S37. In the case of nothaving performed the processing for all the images in the cluster, themain CPU 11 makes a negative decision in the step S36 and causes theflow of control to return to the step S33. When the flow of controlreturns to the step S33, the main CPU 11 specifies another image fileamong the specified image files constituting the cluster and causes theflow of control to proceed to the step S34.

In the step S37, the main CPU 11 makes a decision as to whether or notthe number of the probabilities P of being an event, which are added tothe above Psum, accounts for equal to or greater than N percent of thenumber of all the image files in the cluster. If the number of the addedPs accounts for equal to or greater than N percent, the main CPU 11makes a positive decision in the step S37 and causes the flow of controlto proceed to a step S38. If the number of the added Ps accounts forless than N percent, the main CPU 11 makes a negative decision in thestep S37, and causes the flow of control to proceed to a step S41. Inthe step S41, the main CPU 11 sets Psum to be zero (Psum=0) and causesthe flow of control to proceed to the step S38.

In the step S38, the main CPU 11 makes a decision as to whether or notthe calculation of the Psum has been completed for all the eventcandidates. If the calculation has been completed for all the events,the main CPU 11 makes as positive decision in the step S38 and causesthe flow of control to proceed to a step S39. If the calculation has notbeen done for all the events, the main CPU 11 makes a negative decisionin the step S38 and causes the flow of control to return to the stepS32. When the flow of control returns to the step S32, the main CPU 11specifies another event to be a decision target among the eventcandidates and causes the flow of control to proceed to the step S33.

In the step S39, the main CPU 11 determines the event candidate havingthe maximum Psum among the Psums calculated for each event candidate tobe the title event of the said cluster and causes the flow of control toproceed to a step S42.

In the step S42, the main CPU 11 makes a decision as to whether or notthe calculation of the Psum and the determination of the title eventhave been completed for all the clusters. In the case of havingcompleted the processing for all the clusters, the main CPU 11 makes apositive decision in the step S42 and terminates the flow of control ofFIG. 5. In the case of not having completed the processing for all theclusters, the main CPU 11 makes a negative decision in the step S42 andcauses the flow of control to return to the step S31. When the flow ofcontrol returns to the step S31, the main CPU 11 specifies anothercluster and causes the flow of control to proceed to the step S32.

FIGS. 9 to 11 show examples of event decision for clusters including aset of image files (the number of images=5). The event candidates are,for instance, “Field Day”, “Wedding”, and “Cherry Blossom Viewing”, andaccordingly the probabilities P of being “Field Day”, “Wedding”, and“Cherry Blossom Viewing” are calculated respectively for each of theimage 1 to image 5. The present example assumes that the decisionthreshold Pth=40 percent and that the decision threshold N=40 percent.

In the case of FIG. 9, the Psum for the probability of being “Field Day”is obtained by adding each P of the images in which the probability P ofbeing “Field Day” is equal to or greater than the decision threshold Pth(i.e., image 1, image 2, image 3, and image 5) (step S35). In this case,the Psum is calculated as follows: 85 (image 1)+90 (image 2)+80 (image3)+75 (image 5)=330.

In the case of FIG. 10, the image 4 is the only image in which theprobability P of being “Wedding” becomes equal to or greater than thedecision threshold Pth. Therefore, only the P calculated for the image 4is to be added to obtain the probability Psum of being “Wedding”. As aresult, the Psum=45 (image 4) is obtained. Here, the image 4 (one image)accounts for 20 percent of the total number of images (five), that is,it falls below the above N (40 percent). The main CPU 11 therefore makesa decision that the Psum in the case of FIG. 10 is zero (step S41).

In FIG. 11, the Psum for the probability of being “Cherry BlossomViewing” is obtained by adding each P of the images in which theprobability P of being “Cherry Blossom Viewing” is equal to or greaterthan the decision threshold Pth (i.e., image 1, image 2, image 4, andimage 5) (step S35). In this case, the Psum is calculated as follows: 60(image 1)+70 (image 2)+65 (image 4)+75 (image 5)=270.

The main CPU 11 designates “Field Day”, which corresponds to the maximumPsum, as the title event of the said cluster from among the eventcandidates (“Field Day”, “Wedding”, and “Cherry Blossom Viewing”).

<Cluster Integration Processing >

The cluster integration processing (S40) will now be explained in detailwith reference to the flowchart shown as an example in FIG. 6. In a stepS51 of FIG. 6, the main CPU 11 makes a decision as to whether or not thetime difference between adjacent clusters among the plurality ofclusters is equal to or less than a decision threshold T. If, forexample, the difference between the last shooting time in one clusterand the earliest shooting time in the other cluster is equal to or lessthan the above T, the main CPU 11 makes a positive decision in the stepS51 and causes the flow of control to proceed to a step S52. If the timedifference exceeds the above T, the main CPU 11 makes a negativedecision in the step S51 and terminates the flow of control of FIG. 6.When making a negative decision in the step S51, the main CPU 11 doesnot perform cluster integration.

In the step S52, the main CPU 11 makes a decision as to whether or notthe title events of the adjacent clusters with a time difference equalto or less than T are the same. If the title events are the same, themain CPU 11 makes a positive decision in the step S52 and causes theflow of control to proceed to a step S53. If the title events are notthe same, the main CPU 11 makes a negative decision in the step S52 andterminates the flow of control of FIG. 6. When making a negativedecision in the step S52, the main CPU 11 does not perform clusterintegration.

In the step S53, the main CPU 11 integrates the two clusters into onecluster and terminates the flow of control of FIG. 6. The main CPU 11repeats the cluster integration processing for all the clusters. As aresult of the above processing, the number of clusters is reduced.

<Decision Processing of Representative Image for Each Cluster>

The representative image decision processing in terms of cluster willnow be explained in detail with reference to the flowchart shown as anexample in FIG. 7. In a step S61 of FIG. 7, the main CPU 11 specifiesone cluster from among a plurality of clusters and causes the flow ofcontrol to proceed to a step S62. The specifying order is, for instance,a chronological order with respect to shooting date and time (priorityis given to the cluster having an image file of the earliest shootingtime).

In the step S62, the main CPU 11 reads out selection criterioninformation corresponding to the title event determined in the step S39from the flash memory 19 and causes the flow of control to proceed to astep S63. The selection criterion information, in which decision methodof a representative image is predefined for each title event and tabled,is recorded in the flash memory 19.

In the step S63, based upon the selection criterion information the mainCPU 11 selects a representative image from among an image file groupincluded in the cluster and causes the flow of control to proceed to astep S64. In the step S64, the main CPU 11 mikes a decision as towhether or not it has selected representative images for all theclusters. In the case of having completed the processing for all theclusters, the main CPU 11 makes a positive decision in the step S64 andterminates the flow of control of FIG. 7. In the case of not havingcompleted the processing for all the clusters, the main CPU 11 makes anegative decision in the step S64 and causes the flow of control toreturn to step S61. When the flow of control returns to the step S61,the main CPU 11 specifies another cluster and causes the flow of controlto proceed to the step S62.

The selection criterion information will now be explained. Withreference to the selection criterion information table shown as anexample in FIG. 12, the main CPU 11 selects selection criterioninformation corresponding to the title event of the cluster. Theselection criterion information table is created in advance and recordedin the flash memory 19. If, for instance, the title event of the clusteris “Wedding”, “New Year's Shrine Visit”, “Doll Festival”, “EntranceCeremony”, or “Graduation Ceremony”, the main CPU 11 selects an image inwhich the proportion of the face region included therein is the nearestto a predetermined proportion as a representative image among the imagesincluded in the said cluster. It is to be noted that since facedetection processing, performed based upon image data, to detect the“face” of a person included in an image is a publicly known technique,an explanation will be curtailed now.

In addition, in the event that the title event of a cluster is “SeaBathing”, “Diving”, “Leaf Peeping”, or “Golfing”, the main CPU 11selects an image in which the proportion of a predetermined color regionincluded therein is the nearest to a predetermined proportion as arepresentative image among the images included in the said cluster. Thepredetermined color region is, for instance, a blue region (Sea Bathingand Diving), a red or a yellow region (Leaf Peeping), or a green region(Golfing).

In addition, it may also be arranged that the main CPU 11 selects animage in which the probability P of being an event corresponding to thetitle event of the said cluster becomes the greatest as a representativeimage among the images included in the cluster. In this manner, arepresentative image decision condition is determined for each titleevent in advance, and an image representing the title event isdetermined based upon the decision condition.

According to the first embodiment explained above, the followingoperations and advantageous effects can be achieved.

(1) It is arranged that a plurality of event candidates (field day,wedding, and cherry blossom viewing) are assigned to a cluster (set ofimage files), a characteristic amount appropriate to make a decision asto each of the event candidates is calculated for each image in thecluster, and, based upon the characteristic amount of the calculatedindividual images, the event representing the said cluster (set of imagefiles) is determined from among the event candidates. Unlike theconventional technologies, since a simple addition to calculate the sumPsum of the probabilities P of being an event for each image, a count ofthe number of the images added to the sum Psum, a magnitude comparisonof the Psums between the event candidates, and a magnitude comparisonbetween the decision threshold Pth and the probability P are all thatrequired without applying the SVM method to the cluster, i.e., withoutthe need to calculate the characteristic amount of the cluster, theprocessing can be carried out in a shorter period of time compared tothe case in which the characteristic amount of the image group iscalculated.

(2) Since it is arranged that the event candidate having the maximumvalue of the sum Psum of the probability P of being an event for eachimage is to be determined, an event representing the set of image filescan be determined appropriately.

(3) Since it is arranged that the probability P of being an event foreach image is not to be added to the sum Psum if it falls below thedecision threshold Pth, the extent to which the image with lessprobability of being the event candidate affects the sum Psum can bereduced.

(4) It is arranged that the Psum is made zero so as to exclude the saidevent candidate in the case where the number of images that satisfy thedecision threshold Pth (that is, the number of images in which P isadded to the sum Psum) falls below a predetermined proportion to thenumber of images in the cluster. As a result, an event representing theset of image files can be determined appropriately.

(5) It is arranged that a cluster (set of image files) is prepared byclustering in accordance with the shooting date and time of images. As aresult, an event candidate in accordance with the shooting date and timecan be assigned.

(6) Since it is arranged that a plurality of event candidates areassigned to the cluster, the most appropriate candidate can bedetermined from among the plurality of event candidates. It is to benoted that it may also be arranged that a plurality of event candidatesare assigned to a single image in place of a set of image files and thatthe most appropriate candidate is to be determined from among theplurality of event candidates.

(7) Since it is arranged that the plurality of event candidates areassigned based upon the event candidate table created in advance basedupon the months in which past events took place, an event candidate witha high matching rate can be assigned.

(8) Since it is arranged that the upper limit of the number of eventcandidates corresponding to each month in the event candidate table isthree, load of processing to calculate the probability P of being anevent be reduced more than that in the case of not setting the upperlimit of the number of event candidates. It is to be noted that it mayalso be arranged to allow three or more event candidates correspondingto each month so as to select three high-priority candidates whenassigning the event candidates to the cluster.

(9) It is arranged that adjacent clusters are integrated into one if thetime difference between the adjacent clusters among the plurality ofclusters is equal to or less than the decision threshold T and the titleevents of the said clusters are the same. As a result, compared to thoseclustered only by the shooting date and time, clustering in detail morethan necessary can be avoided more, thereby cataloging the images asdesired.

(10) It is arranged that a positive decision is made in the step S51 ifthe difference between the last shooting time in one cluster and theearliest shooting time in the other cluster is equal to or less that theabove T. Since images of the same event are often captured seriallywithout intervals, it is possible to detect the case in which the titleevents of the said clusters are highly likely to be the same.

(11) It is arranged that the relation between the event candidate andthe characteristic amount to be calculated is tabled in advance andrecorded in the flash memory 19. As a result, the characteristic amountof the image appropriate to make a decision as to event candidates canbe calculated, so that an event representing the set of image files canbe determined appropriately.

(12) it is arranged that, based upon selection criterion informationprovided corresponding to the title event of the cluster, arepresentative image is selected from among images included in the saidcluster. As a result, the representative image can be determined byusing the most appropriate selection criterion for each cluster.

(13) It is arranged that selection criterion information appropriate foreach title event is tabled in advance and recorded in the flash memory19. In the case of different title event, selection criterion isswitched with reference to the said selection criterion informationtable. By tabling the selection criterion information, combinations ofthe title event and the selection criterion can be arbitrarily set.

(Variation 1)

It may also be arranged that clustering is performed in terms ofshooting date.

(Variation 2)

It may also be arranged that clustering is performed not in accordancewith shooting date and time information but in accordance with shootingposition information. More specifically, in the step S11 (FIG. 3), themain CPU 11 extracts from all the image files positioning informationrepresenting the shooting position recorded in the additionalinformation unit of the image file. Then, in the step S13 (FIG. 3), withone cluster as a starting point of processing for each image, the mainCPU 11 groups image files into a plurality of clusters (sets of imagefiles with the similar shooting position) by repeating processing tosequentially integrate clusters with the similar shooting position. Inthe event that the interval (distance) between the shooting positions inadjacent clusters is equal to or greater than a predetermined distance(for instance, 1 Km), the main CPU 11 causes the flow of control toproceed to the step S14.

(Variation 3)

It may also be arranged that the event candidate table shown as anexample in FIG. 8 can be modified by a user operation. For instance, theuser operates the operation members 17 with an edit screen of the eventcandidate table being displayed on the LCD monitor 16 so as to modifythe table content. The modification content is recorded in the flashmemory 19. It is to be noted that the number of event candidatesassigned to each month is as described above, preferably and normallylimited to a predetermined number (for instance, three).

(Variation 4)

It may also be arranged that the step S22 of FIG. 4 is skipped. In thiscase, all the event candidates included in the system are selectedwithout reference to the event candidate table.

(Variation 5)

It is preferable that, in the event candidate table in the case ofvariation 2, event candidates are arranged to correspond to, forexample, each area including the shooting position. For instance, anevent such as sea bathing, watermelon splitting, surfing, or diving isselected if the area is near the sea. An event such as camping or hikingis selected if the area is near the mountain. The event candidate tablein this case, including events that take place frequently in each area,is created in advance based upon the areas in which past events tookplace and recorded in the flash memory 19.

(Variation 6)

It may also be arranged that event candidates are assigned in accordancewith shooting time. In this case, for example, “New Year's Party”, “YearEnd Party”, “Banquet”, and the like are included in the event candidatein which images are captured after 18 o'clock. On the other hand, theevent candidate does not include those do not take place after 18o'clock based upon the time in which past events took place such as“Field Day”, “Golfing”, “Excursion”, “Diving”, and the like. By reducingthe number of event candidates, load of processing to calculate theprobability P of being an event for each image can be reduced.

(Variation 7)

It may also be arranged that the title event of the cluster (set ofimage files) is determine in accordance with the number of images inwhich the probability P of being an event exceeds the decision thresholdPth. For example, among event candidates (“Field Day”, “Wedding”, and“Cherry Blossom Viewing”), the event candidate corresponding to the oneincluding the greatest number of images exceeding the decision thresholdPth is determined to be the title event of the said cluster.

(Variation 8)

In addition, the determination method to determine the event candidateincluding the greatest number of images in which the probability P ofbeing an event exceeds the decision threshold Pth to be the title eventand the determination method to determine the event candidatecorresponding to the maximum Psum to be the title event may be combined.In this case, if the title event of the cluster can not be determinedusing one of the determination method, a decision is made using theother determination method. For example, in the examples of FIG. 9 toFIG. 11, the numbers of images in which the probability P of being anevent exceeds the decision threshold Pth are the same in the eventcandidates “Field Day” and “Cherry Blossom Viewing” and accordingly thetitle event of the cluster can not be determined. Therefore, the eventcandidate “Field Day” corresponding to the maximum Psum is determined tobe the title event of the said cluster.

(Variation 9)

When a decision is made as to whether or not the time difference betweenadjacent clusters is equal to or less than the decision threshold T, adecision may also be made based upon the time difference correspondingto the centroid of each cluster. In this case, if the difference betweenthe average shooting time of the image group belonging to one clusterand that of the image group belonging to the other cluster is equal toor less than the above T, the main CPU 11 makes in the step S51 (FIG. 6)a positive decision in the step S51 and causes the flow of control toproceed to the step S52.

(Variation 10)

In the case of variation 2 and variation 5, a decision is made as towhether or not the distance between adjacent clusters is equal to orless than the decision threshold D. In this case, if the shortestdistance of those between shooting positions of the image groupbelonging to one cluster and shooting positions of the image groupbelonging to the other cluster is equal to or less than the above D, themain CPU 11 makes in the step S51 (FIG. 6) a positive decision in thestep S51 and causes the flow of control to proceed to step S52. Sinceimages of the same event are often captured serially in the same area,it is possible to detect the case in which the title events of the saidclusters are highly likely to be the same.

According to variation 10, among the plurality of clusters, adjacentclusters are integrated into one if the difference in distance betweenthe adjacent clusters is equal to or less than the decision threshold Dand the title events of the said clusters are the same. As a result,compared to those clustered only by the shooting position, clustering indetail more than necessary can be avoided more, thereby cataloging theimages as desired.

(Variation 11)

While the example in which images are grouped in the electronic camera 1and their titles are assigned has been explained, an image titleassigning device may also be arranged by causing a computer device 10shown in FIG. 13 to execute a title assigning program to perform theprocessing shown in FIG. 2 to FIG. 7. In order to use the titleassigning program loaded into the personal computer 10, the program isloaded into a data storage device of the personal computer 10 and thesaid program is executed so as to use the personal computer 10 as animage group title assigning device.

The load of the program into the personal computer 10 may be performedusing a recording medium 104, such as a CD-ROM in which the program isstored, being set to the personal computer 10 or may be performed via acommunication line 101 such as a network. If the load is performed viathe communication line 101, the program is to be stored in a hard diskdevice 103 or the like of a server (computer) 102 connected to thecommunication line 101. The title assigning program can be supplied as acomputer program product in a variety of forms such as via the recordingmedium 104 or the communication line 101.

In addition, it may also be arranged that the program is executed by theserver side in a form as the ASP (Application Service Provider).

(Variation 12)

It may also be arranged that a multidimensional event candidate table isprovided by combining at least two of the conditions according to whichthe event candidate is assigned, i.e., the shooting date and timeinformation, the shooting position information, the shooting condition,and the presence or absence of a specific subject, which are describedabove.

(Variation 13)

It may also be arranged that the selection criterion information tablecan be modified by a user operation. For instance, the user operates theoperation members 17 with an edit screen of the selection criterioninformation table being displayed on the LCD monitor 16 so as to modifythe table content. The modification content is recorded in the flashmemory 19.

Second Embodiment

It may also be arranged that clustering processing (S10) is performednot in accordance with shooting date and time information but inaccordance with shooting position information. More specifically, in thestep S11 (FIG. 3), the main CPU 11 extracts from all the image filespositioning information representing the shooting position recorded inthe additional information unit of the image file. Then, in the step S13(FIG. 3), with one cluster as a starting point of processing for eachimage, the main CPU 11 groups image files into a plurality of clusters(sets of image files with the similar shooting position) by repeatingprocessing to sequentially integrate clusters with the similar shootingposition. If the interval (distance) between the shooting positions inadjacent clusters is equal to or greater than a predetermined distance(for instance, 1 Km), the main CPU 11 causes the flow of control toproceed to the step S14.

In the event candidate table in the case of the second embodiment, eventcandidates are arranged to correspond to each area including theshooting position. FIG. 14 shows an example of the event candidate tablereferred to in the second embodiment. For instance, in the case of Kantoregion, a plurality of event candidates are arranged to correspond toeach of Tokyo Metropolis, Kanagawa Prefecture, Chiba Prefecture, SaitamaPrefecture, Gunma Prefecture, Ibaraki Prefecture, and TochigiPrefecture. The event candidates include the names of the area, placesof interest, parks, theme parks, landmarks, and the like. With referenceto the event candidate table, the main CPU 11 selects an eventcorresponding to the area (Metropolis or Prefectures) indicated by theshooting position information of an image file constituting the cluster.For instance, if the image file is captured in Chiba Prefecture, themain CPU 11 renders “Disney Resort”, “Narita International Airport”,“Mother Farm (registered trademark)”, and the like event candidates. Theevent candidate table, including events that take place frequently ineach area, is created in advance based upon the places in which pastevents took place and recorded in the flash memory 19. In addition, thenumber of event candidates arranged to correspond to each of the areas(in the present example, Metropolis and Prefectures) is limited to apredetermined number (in the present example, three).

The main CPU 11 makes a decision as to whether or not the distancebetween adjacent clusters is equal to or less than the decisionthreshold D. More specifically, if the shortest distance of thosebetween shooting positions of the image group belonging to one clusterand shooting positions of the image group belonging to the other clusteris equal to or less than the above D, the main CPU 11 makes in the stepS51 (FIG. 6) a positive decision in the step S51 and causes the flow ofcontrol to proceed to step S52. Since images of the same event are oftencaptured serially in the same area, it is possible to detect the case inwhich the title events of the said clusters are highly likely to be thesame.

According to the second embodiment explained above, among the pluralityof clusters, adjacent clusters are integrated into one if the differencein distance between the adjacent clusters is equal to or less than thedecision threshold D and the title events of the said clusters are thesame. As a result, compared to those clustered only by the shootingposition, clustering in detail more than necessary can be avoided more,thereby cataloging the images as desired.

In addition, since it is arranged that the plurality of event candidatesare assigned based upon the event candidate table created in advancebased upon the places in which past events took place, an eventcandidate with a high matching rate can be assigned.

Furthermore, in the same manner as the first embodiment, since it isarranged that the upper limit of the number of event candidatescorresponding to each area in the event candidate table is three, loadof processing to calculate the probability P of being an event bereduced more than that in the case of not selling the upper limit of thenumber of event candidates.

(Variation 14)

It may also be arranged that event candidates are assigned in accordancewith shooting condition in place of shooting date and time informationand shooting position information. In this case, an event candidatetable in which event candidates are arranged to correspond to eachshooting condition is provided. For instance, shooting conditions(shutter speed, aperture, focus control information, flash firing or notfiring, color temperature adjustment factor, and the like) having beenstored in additional information data are used so that “Car Racing”,“Motorcycle Speedway”, “Sport Competition”, and the like are included inthe event candidates in the case where the shutter speed is higher thana predetermined value. In addition, “Wedding”, “Christmas”, “Banquet”,and the like are included in the event candidates in the case whereflash was fired. On the other hand, those events that were photographedwithout flash firing when they took place, such as “Skiing”, “CherryBlossom Viewing”, “Excursion”, and “Sea Bathing”, are not included inthe event candidates. By reducing the number of event candidates, loadof processing to calculate the probability P of being an event for eachimage can be reduced.

(Variation 15)

It may also be arranged that an event candidate is assigned inaccordance with the presence or absence of a specific subject in theimage. For example, face detection processing is performed based uponthe image data, and if the “face” of the person is included in theimage, “New Year's Party”, “Graduation Ceremony”, “Entrance Ceremony”,“Year End Party”, and the like are included in the event candidates. Onthe other hand, those events that do not include the “face” on theirimages captured when they took place, such as “Leaf Peeping”, “Diving”,and the like, are not included in the event candidates. By reducing thenumber of event candidates, load of processing to calculate theprobability P of being an event for each image can be reduced.

(Variation 16)

In addition, if it is arranged to identify the “face” of a specificperson and the “face” of the specific person such as a family member isincluded in an image, “New Year's Shrine Visit”, “Seven-Five-ThreeFestival” (festival for children of specific ages), “Doll Festival”, andthe like may preferably be included in the event candidates. In thiscase, an occasion in which the family member does not participate is notincluded in the event candidates. By reducing the number of eventcandidates, load of processing to calculate the probability P of beingan event for each image can be reduced. It is to be noted that sinceface identification processing to identify the “face” of a personincluded in an image is a publicly known technique, an explanation willbe curtailed now.

(Variation 17)

Furthermore, it may also be arranged to make a decision as to whether ornot the “face” of the person is that of an adult or that of a child andthat of a male or that of a female and to include event candidatesmatching each event or exclude those not matching each event. Byreducing the number of event candidates, load of processing to calculatethe probability P of being an event for each image can be reduced.

(Variation 18)

It is to be noted that it may also be arranged that a multidimensionalevent candidate table is provided by combining at least two of thefollowing conditions according to which the event candidate is assigned,i.e., the shooting date and time information, the shooting positioninformation, the shooting condition, and the presence or absence of aspecific subject, which are explained above.

Third Embodiment

In place of the selection criterion information table shown as anexample in FIG. 12, the selection criterion information table shown asan example in FIG. 15 may be referred to. The selection criterioninformation table of FIG. 15 is, in the same manner as the tableaccording to the first embodiment, created in advance and recorded inthe flash memory 19. For example, if the title event of a cluster is“Trip to Hokkaido”, the main CPU 11 selects an image in which theshooting position information is the nearest to position informationcorresponding to the said area (for instance, places of interest inHokkaido) as a representative image.

More specifically, the position information related to the name of thearea included in the title event is recorded in the flash memory 19 inadvance. For instance, if “Hokkaido” is included in the title event, themain CPU 11 reads out position information corresponding to places ofinterest (for example, Sarobetsu Mire) of “Hokkaido” from the flashmemory 19. In addition, the main CPU 11 makes a decision as to whetheror not position information of the shooting point is recorded inadditional information data of each image file of all the image filesincluded in the cluster, and, if the position information of theshooting point is recorded, obtains the position information from thesaid image file.

Then, based upon the position information read out from the flash memory19 and the position information obtained from the image file in thecluster, the main CPU 11 makes a decision as to whether or not each ofthe images has been captured in any of the places of interest. As aresult of comparison, the main CPU 11 selects the image in which thedistance between the both position information is the shortest as arepresentative image of the cluster.

In addition, if the title event of a cluster is “Diving” or “SeaBathing”, the main CPU 11 selects “the image in which the sea isphotographed”, i.e., the image in which the proportion of a specificcolor region (in this case, the sea) in the image is the nearest to apredetermined proportion as a representative image. More specifically,color information related to the title event is recorded in the flashmemory 19 in advance. For instance, if characters associated with “Sea”are included in the title event, the main CPU 11 reads out informationshowing a specific color (for example, cobalt blue) from the flashmemory 19.

In addition, the main CPU 11 obtains color information from image dataof each image file of all the image files included in the cluster. Forexample, the main CPU 11 divides an image into a predetermined number aregions and makes a decision as to whether or not the color of a pixelgroup positioned at the center of each of the divided regions matchesthe specific color. Then, the main CPU 11 obtains the proportion of theabove predetermined number and the number of the divided regionsmatching the specific color. The proportion of the specific color (thecolor of the sea) accounting for in an image varies depending upon thescene, such as a shooting scene on a ship, a shooting scene on thebeach, a shooting scene in water, or the like. Therefore, the proportioninformation in accordance with the scene desired to be a representativeimage is recorded in the flash memory 19 in advance. In this manner,information showing processing necessary to determine a representativeimage and also information necessary for the said processing arerecorded in the flash memory 19.

Furthermore, if the title event of a cluster is “Field Day”, the mainCPU 11 selects an image in which main subject movement information isgreater than a predetermined value as a representative image. Morespecifically, information that instructs acquisition of subject trackinginformation is recorded in the flash memory 19 in advance. For instance,if the title event includes characters associated with “Sports” such as“Field Day”, the main CPU 11 reads out the information that instructsacquisition of the subject tracking information from the flash memory19.

In addition, the main CPU 11 makes a decision as to whether or notinformation related to the subject tracking result in additionalinformation data of each image file of all the image files included inthe cluster is recorded, and, tithe information related to the subjecttracking result is recorded, obtains the information related to thesubject tracking result from the said image file.

Then, based upon the obtained tracking information, the main CPU 11compares the time series variations in coordinate value of the subjectposition between each of the images. As the result of the comparison,the main CPU 11 selects the image with the greatest variation incoordinate value of the subject position as a representative image ofthe cluster.

When providing the user with the folder structure corresponding to eachcluster, the main CPU 11 causes the LCD monitor 16 to display athumbnail image of a representative image, for example, arranged on afolder icon.

More specifically, the main CPU 11 causes a thumbnail image of therepresentative image in which places of interest are captured to bedisplayed on the folder icon of the title event of “Trip to Hokkaido”.In addition, the main CPU 11 causes a thumbnail image of therepresentative image in which an area in cobalt blue accounts for apredetermined proportion of the captured image to be displayed on thefolder icon of the title event of “Sea Bathing”. Furthermore, the mainCPU 11 causes a thumbnail image of the representative image in which themain subject with great movement is captured to be displayed on thefolder icon of the title event of “Field Day”.

According to the third embodiment explained above, the followingoperations and advantageous effects can be achieved.

(1) It is arranged that, if the title event includes the name of an areaXXX such as “Trip to XXX”, an image having shooting position informationthat is the nearest to a predetermined point (for example, a place ofinterest or the like) in the area is selected as a representative image.Since in general a place of interest is known to represent an area, apreferred image as a representative of a plurality of imagesconstituting a cluster can be determined.

(2) It is arranged that, if the title event is associated with aspecific color such as “Sea Bathing”, an image in which the subjectregion of the color in the image accounts for a predetermined proportionis selected as a representative image. Since in general cobalt blue isknown to represent the color of the sea, if cobalt blue is associatedwith “Sea Bathing” in advance, a preferred image as a representative ofa plurality of images constituting a cluster can be determined.

(3) It is arranged that, in the event that the title event includes amoving subject such as “Field Day”, an image with a great variation incoordinate indicating the subject position is selected as arepresentative image. Since in general a field day is known as an eventwith many moving scenes such as a foot race, a preferred image as arepresentative of as plurality of images constituting a cluster can bedetermined.

(Variation 19)

It may also be arranged to record position information on a plurality ofplaces of interest (for instance, Sarobetsu Mire and Rishiri Island) inthe flash memory 19 in advance and to prioritize the places of interest(for example, (1) Rishiri Island and (2) Sarobetsu Mire), in place ofselecting an image with the shortest distance from a place of interestas a representative image based upon shooting position information. Inthe event that images captured at a plurality of different places ofinterest are included among images constituting the cluster, the imagehaving position information corresponding to the place of interest withthe highest priority (for example, Rishiri Island) is selected as arepresentative image from among images captured within a predetermineddistance (for instance, 500 m) from the places of interest. As a result,a representative image according to user preferences can be determinedin the case where captured images for a plurality of places of interestare included.

(Variation 20)

If the title event of the cluster is “Diving” or “Sea Bathing”, an imagein which the proportion of the sea thereof is a predetermined value andthe focal length of the taking lens at the time of shooting is shorterthan a predetermined value may also be selected as a representativeimage. Information of shooting condition stored as additionalinformation data of the said image file is used for focal lengthinformation. As a result, a representative image with a wide pictureangle can be selected.

Fourth Embodiment

FIG. 16 is a block diagram showing the structure of an embodiment of theimage display device of the present embodiment. An image display device100, for example a personal computer, includes an operation member 101,a connection IF (interface) 102, a control device 103, as HDD (hard diskdrive) 104, and a monitor 105.

The operation member 101 includes a variety of devices to be operated bythe user, for instance, a keyboard and a mouse. A USB interface to allowwired connection with, for instance, a digital camera or a video camera,a wireless LAN module to allow wireless connection, or the like ismounted as the connection IF 102, which is an interface to connect anexternal device such as a digital camera. In the present embodiment, animage file is loaded from the digital camera through the connection IF102. The monitor 105, for example a liquid crystal monitor, displaysdata to be displayed output from the control device 103.

The HDD 104 is a recording device to record image files loaded throughthe connection IF 102, a variety of programs executed by the controldevice 103, and the like. In the present embodiment, it is assumed thatimage files loaded through the connection IF 102 have been classifiedaccording to the attributes of the images in advance. It is assumedthat, for example, they have been grouped and classified by shootingyear, month, and day or grouped and classified by event. It is assumedthat they have then been sorted by each group into a folder and recordedin the HDD 104.

FIG. 17 is a schematic illustration of an example of the folderstructure in the present embodiment. The example shown in FIG. 17presents a case in which the image files are sorted into folders byshooting year, month, and day. More specifically, FIG. 17 presents anexample in which a folder 2 a in which an image file captured on Dec. 1,2007 is recorded, a folder 2 b in which an image file captured on Dec.2, 2007 is recorded, and a folder 2 c in which an image file captured onDec. 5, 2007 is recorded are created in the HDD 104.

The control device 103, being constituted with a CPU, a memory, andother peripheral circuits, functionally includes a subject recognitionunit 103 a and a representative image selection unit 103 b. It is to benoted that the memory constituting the control device 103 includes anSDRAM and a flash memory. An SDRAM is a volatile memory that is used asa work memory so that the CPU develops a program when executing theprogram or used as a buffer memory in which data is temporarilyrecorded. The flash memory is a non volatile memory in which data of theprogram to be executed by the control device 103, a variety ofparameters to be read when the program is executed, and the like arerecorded.

In the present embodiment, when presenting the folder structure createdin the HDD 104 as described above to the user, the control device 103causes a representative image selected from among image files(hereinafter referred to as “images”) recorded in each of the folders tobe associated with the folder and to be displayed. For instance, whenproviding the user with the folder structure shown in FIG. 17, thecontrol device 103 causes a thumbnail image of the representative to bearranged on the folder icon as shown in FIG. 18 and to be displayed. Inorder to do so, the subject recognition unit 103 a and therepresentative image selection unit 103 b carry out the processing shownin flowcharts of FIG. 19 to FIG. 21.

It is to be noted that the processing shown in FIG. 19 is carried out bythe control device 103 as a program that runs when display of a folderlist on the monitor 105 is instructed by the user operating theoperation member 101.

In a step S10, the subject recognition unit 103 a carries out publiclyknown subject recognition processing for all the images recorded in anyone of the folders in the HDD 104. As the result, the subjectrecognition unit 103 a can recognize what has been captured as thesubject for each image recorded in the target folder. It is to be notedthat if the face of a person is captured in an image, the coordinatevalue of the face outline in the image is specified in the publiclyknown subject recognition processing. In addition, whether or not theface of the person is facing the front or how many degrees it turns awayfrom the front is also detected. The subject recognition unit 103 arecords coordinate value information of the face outline and informationrelated to the face direction in a RAM.

After that, the control device 103 causes the flow of control to proceedto a step S20, in which, as the result of the subject recognitionprocessing performed in the step S10, the representative image selectionunit 103 b makes a decision as to whether or not an image with a personcaptured as the subject therein has been included in the target folder.If the representative image selection unit 103 b makes a positivedecision, the control device 103 causes the flow of control to proceedto a step S30, in which the representative image selection unit 103 bcarries out “representative image selection processing in the case of aperson photographed” shown in FIG. 19 and selects a representative imagefrom among images recorded in the target folder, and the control device103 causes the flow of control to proceed to a step S50. On the otherhand, if the representative image selection unit 103 b makes a negativedecision, the control device 103 causes the flow of control to proceedto a step S40. In the step S40, the representative image selection unit103 b carries out “representative image selection processing in the caseof a person not photographed” shown in FIG. 20 and selects arepresentative image from among images recorded in the target folder,and the control device 103 causes the flow of control to proceed to thestep S50.

In other words, based upon the result of the subject recognition by thesubject recognition unit 103 a, the representative image selection unit103 b changes the processing to select the representative image. Here,the processing shown in FIG. 19 and FIG. 20 have, as described later,different selection criteria to select the representative image.Therefore, in other words, based upon the result of the subjectrecognition by the subject recognition unit 103 a, the representativeimage selection unit 103 b sets selection criteria to select therepresentative image.

In the step S50, the representative image selection unit 103 b makes adecision as to whether or not the execution of the processing from thestep S10 to the step S40 has been completed for all the folders recordedin the HDD 104. If the representative image selection unit 103 b makes anegative decision, the control device 103 causes the flow of control toreturn to the step S10, in which the representative image selection unit103 b designates another folder as a new target folder and repeats theprocessing. On the other hand, in the event that the representativeimage selection unit 103 b makes a positive decision, the control device103 causes the flow of control to proceed to a step S60. In the stepS60, the representative image selection unit 103 b displays, as shown inFIG. 18, the selected representative image arranged in the folder icon,and the control device 103 terminates the flow of control.

FIG. 20 is a flowchart showing the flow of the “representative imageselection processing in the case of a person photographed” carried outin the step S30 of FIG. 19. In a step S110, the representative imageselection unit 103 b makes a decision as to whether or not an image inwhich a family member or an acquaintance is captured has been includedin the target folder. More specifically, an face image of a familymember or an acquaintance is captured and recorded in the HDD 104 inadvance, and then, the representative image selection unit 103 bperforms processing to match the face image recorded in the HDD 104 tothe image in which the person is captured in the target folder so as tomakes a decision as to whether or not the face in the image is the faceof the family member or the acquaintance.

If the representative image selection unit 103 b makes a positivedecision in the step S110, the control device 103 causes the flow ofcontrol to proceed to a step S120, in which the representative imageselection unit 103 b makes a decision as to whether or not only assingle image in which the family member or the acquaintance is capturedhas been included in the target folder. If the representative imageselection unit 103 b makes a positive decision, the control device 103causes the flow of control to proceed to a step S130, in which therepresentative image selection unit 103 b selects the single imagespecified as an image in which the family member or the acquaintance iscaptured in the decision processing of the step S120 as a representativeimage, and causes the flow of control to return to the processing shownin FIG. 19.

On the other hand, if the representative image selection unit 103 bmakes a negative decision in the step S120, i.e., if the representativeimage selection unit 103 b makes a decision that a plurality of imagesin which the family member or the acquaintance is captured have beenincluded in the target folder, the control device 103 causes the flow ofcontrol to proceed to a step S140. In the step S140, the representativeimage selection unit 103 b makes a decision as to whether or not animage in which a face is facing the front has been included among theimages in which the family member or the acquaintance is captured. Morespecifically, based upon information related to the direction of theface detected and recorded in the RAM by the subject recognition unit103 a in the step S10 of FIG. 19, the representative image selectionunit 103 b makes a decision that the face is facing the front when thedirection of the face is at zero degrees or when it is within apredetermined range from zero degrees, for example, 10 degrees or lessright and left.

If the representative image selection unit 103 b makes a negativedecision in the step S140, the control device 103 causes the flow ofcontrol to proceed to a step S180, in which the representative imageselection unit 103 b specifies the image with the greatest area of theface from among the plurality of images in which the family member orthe acquaintance is captured. More specifically, based upon thecoordinate value information of the face outline detected and recordedin the RAM by the subject recognition unit 103 a in the step S10 of FIG.19, the representative image selection unit 103 b specifies the regioncovered by the face in the image, calculates its area, and, based uponthe calculated result, specifies the image in which the area of the faceis the greatest. After that, the control device 103 causes the flow ofcontrol to proceed to the step S130 described above, in which therepresentative image selection unit 103 b selects the image in which thearea of the face is the greatest specified in the step S180 as arepresentative image, and causes the flow of control to return to theprocessing shown in FIG. 19.

On the other hand, if the representative image selection unit 103 bmakes a positive decision in the step S140, the control device 103causes the flow of control to proceed to a step S150. In the step S150,the representative image selection unit 103 b makes a decision as towhether or not only a single image in which a face is facing the fronthas been included among the images in which the family member or theacquaintance is captured. If the representative image selection unit 103b makes a positive decision, the control device 103 causes the flow ofcontrol to proceed to the step S130 described above, in which therepresentative image selection unit 103 b selects the single imagespecified as an image in which the face is facing the front in thedecision processing. of the step S150 as a representative image, andcauses the flow of control to return to the processing shown in FIG. 19.

On the other hand, if the representative image selection unit 103 bmakes a negative decision in the step S150, i.e., if the representativeimage selection unit 103 b makes a decision that a plurality of imagesin which the face is facing the front have been included, the controldevice 103 causes the flow of control to proceed to a step S160. In thestep S160, the representative image selection unit 103 b makes adecision as to whether or not an image in which the expression of theface facing the front is smile has been included among the images inwhich the lace is facing the front. More specifically, therepresentative image selection unit 103 b carries out publicly knownexpression recognition processing for the plurality of images in whichthe family member or the acquaintance is captured and the face is facingthe front so as to make a decision as to whether or not the face facingthe front is a smile.

If the representative image selection unit 103 b makes a negativedecision in the step S160, the control device 103 causes the flow ofcontrol to proceed to the step S180 described above, in which therepresentative image selection unit 103 b specifies the image in whichthe area of the face is the greatest from among the images in which thefamily member or the acquaintance is captured and the face is facing thefront. After that, the control device 103 causes the flow of control toproceed to the step S130 described above, in which the representativeimage selection unit 103 b selects the image in which the area of theface is the greatest specified in the step S180 as a representativeimage, and causes the flow of control to return to the processing shownin FIG. 19.

On the other hand, if the representative image selection unit 103 bmakes a positive decision in the step S160 the control device 103 causesthe flow of control to proceed to a step S170, in which therepresentative image selection unit 103 b makes a decision as to whetheror not only a single image in which the expression is a smile has beenincluded among the images in which the family member or the acquaintanceis captured and the face is facing the front. If the representativeimage selection unit 103 b makes a negative decision, the control device103 causes the flow of control to proceed to the step S180 described,above, in which the representative image selection unit 103 b specifiesthe image in which the area of the face is the greatest from among theimages in which the family member or the acquaintance is captured, theface is facing the front, and the expression is a smile. After that, thecontrol device 103 causes the flow of control to proceed to the stepS130 described above, in which the representative image selection unit103 b selects the image in which the area of the face is the greatestspecified in the step S180 as a representative image, and causes theflow of control to return to the processing shown in FIG. 19.

On the other hand, if the representative image selection unit 103 bmakes a positive decision in the step S170, the control device 103causes the flow of control to proceed to the step S130 described above,in which the representative image selection unit 103 b selects thesingle image specified as an image in which the expression is a smile inthe decision processing of the step S170 as a representative image, andcauses the flow of control to return to the processing shown in FIG. 19.

Next, the processing when the representative image selection unit 103 bmakes a negative decision in the step S110, i.e., the processing when animage in which the family member or the acquaintance is captured has notbeen included in the folder will be explained. In this case, the controldevice 103 causes the flow of control to proceed to a step S190, inwhich, based upon the information related to the direction of the facerecorded in the RAM, as described above, the representative imageselection unit 103 b makes a decision as to whether or not an image inwhich the face is facing the front has been included among the images inwhich the person is captured.

If the representative image selection unit 103 b makes a negativedecision, the control device 103 causes the flow of control to proceedto the step S180 described above, in which the representative imageselection unit 103 b specifies the image in which the area of the faceis the greatest from among the image in which a person other than thefamily member nor the acquaintance is captured. After that, the controldevice 103 causes the flow of control to proceed to the step S130described above, in which the representative image selection unit 103 bselects the image in which the area of the face is the greatestspecified in the step S180 as a representative image, and causes theflow of control to return to the processing shown in FIG. 19.

On the other hand, if the representative image selection unit 103 bmakes a positive decision in the step S190, the control device 103causes the flow of control to proceed to a step S200. In the step S200,the representative image selection unit 103 b makes a decision as towhether or not only a single image in which the person other than thefamily member or other than the acquaintance is lacing the front hasbeen included in the target folder. If the representative imageselection unit 103 b makes a positive decision, the control device 103causes the flow of control to proceed to the step S130 described above,in which the representative image selection unit 103 b selects thesingle image specified an image in which the face is facing the front inthe decision processing of the step S200 as a representative image, andcauses the flow of control to return to the processing shown in FIG. 19.

On the other hand, if the representative image selection unit 103 bmakes a negative decision in the step S200, i.e., if a plurality ofimages in which the face is facing the front have been included, thecontrol device 103 causes the flow of control to proceed to the stepS180 described above, in which the representative image selection unit103 b specifies the image in which the area of the face is the greatestfrom among the plurality of images in which the person other than thefamily member nor the acquaintance is captured and the face is facingthe front. After that, the control device 103 causes the flow of controlto proceed to the step S130 described above, in which the representativeimage selection unit 103 b selects the image in which the area of theface is the greatest specified in the step S180 as a representativeimage, and causes the flow of control to return to the processing shownin FIG. 19.

FIG. 21 is a flowchart showing the flow of the “representative imageselection processing in the case of a person not photographed” carriedout in the step S40 of FIG. 19. In a step S210, the representative imageselection unit 103 b makes a decision as to whether or not an imagewithout blurring has been included in the target folder. Morespecifically, the representative image selection unit 103 b carries outpublicly known amount of blur calculation processing for all the imagesin the target folder so as to calculate an amount of blur for each ofthe images. Then, the representative image selection unit 103 b makes adecision that the image whose calculated amount of blur is equal to orless than a threshold value is an image without blurring.

In the event that the representative image selection unit 103 b makes apositive decision in the step S210, the control device 103 causes theflow of control to proceed to a step S220, in which the representativeimage selection unit 103 b makes a decision as to whether or not only asingle image without blurring has been included in the target folder. Inthe event that the representative image selection unit 103 b makes apositive decision, the control device 103 causes the flow of control toproceed to a step S230, in which the representative image selection unit103 b selects the single image specified as an image without blurring inthe decision processing of the step S220 as a representative image, andcauses the flow of control to return to the processing shown in FIG. 19.

On the other hand, in the event that the representative image selectionunit 103 b makes a negative decision in the step S220, i.e., in theevent that the representative image selection unit 103 b makes adecision that a plurality of images without blurring have been includedin the target folder, the control device 103 causes the flow of controlto proceed to a step S240. In the step S240, the representative imageselection unit 103 b makes a decision as to whether or not an image inwhich a main subject is focused on has been included among the imageswithout blurring. More specifically, based upon range findinginformation recorded in the image file, the representative imageselection unit 103 b specifies the subject existing in a focus area asthe main subject. Then, based upon a defocus amount of the photographicoptical system recorded in the image file, the representative imageselection unit 103 b makes a decision as to whether or not the mainsubject is in focus. It is to be noted that it is assumed that rangefinding information and the defocus amount of the photographic opticalsystem have been recorded in the image file as additional information,for instance, Exif information.

If the representative image selection unit 103 b makes a negativedecision in the step S240, the control device 103 causes the how ofcontrol to proceed to a step S280, in which the representative imageselection unit 103 b specifies one image captured in the optimalcomposition front among the images without blurring. Here, the imagecaptured in the optimal composition refers to an image captured in amethod generally said to be the optimal composition, for example, therule of thirds. In other words, the representative image selection unit103 b specifies an image in which the position of the main subject,which has been specified in the step S240, in the image is the nearestto the optimal subject position in the rule of thirds as the imagecaptured in the optimal composition. After that, the control device 103causes the flow of control to proceed to the step S230 described above,in which the representative image selection unit 103 b selects the imagecaptured in the optimal composition specified in the step S280 as arepresentative image, and causes the flow of control to return to theprocessing shown in FIG. 19.

On the other hand, if the representative image selection unit 103 bmakes a positive decision in the step S240, the control device 103causes the flow of control to proceed to a the step S250. In the stepS250, the representative image selection unit 103 b makes a decision asto whether or not only a single image in which the main subject isfocused on has been included in the images without blurring. If therepresentative image selection unit 103 b makes a positive decision, thecontrol device 103 causes the flow of control to proceed to the stepS230 described above, in which the representative image selection unit103 b selects the single image specified as an image in which the mainsubject is focused on in the decision processing of the step S250 as arepresentative image, and causes the flow of control to return to theprocessing shown in FIG. 19.

On the other hand, if the representative image selection unit 103 bmakes a negative decision in the step S250, i.e., if the representativeimage selection unit 103 b makes a decision that a plurality of imagesin which the main subject is focused on have been included, the controldevice 103 causes the flow of control to proceed to a step S260. In thestep S260, the representative image selection unit 103 b makes adecision as to whether or not an image with proper brightness has beenincluded among the images in which the main subject is focused on. Morespecifically, the representative image selection unit 103 b creates ahistogram representing a distribution of the brightness value for eachof the plurality of images without blurring in which the main subject isfocused on. Then, based upon the distribution of the brightness valuerepresented by the created histogram, the representative image selectionunit 103 b makes a decision as to whether or not each of the images hasproper brightness.

In the event that the representative image selection unit 103 b makes anegative decision in the step S260, the control device 103 causes theflow of control to proceed to the step S280 described above, in whichthe representative image selection unit 103 b specifies an imagecaptured in the optimal composition described above from among theplurality of images without blurring in which the main subject isfocused on. After that, the control device 103 causes the flow ofcontrol to proceed to the step S230 described above, in which therepresentative image selection unit 103 b selects the image captured inthe optimal composition specified in the step S280 as a representativeimage, and causes the flow of control to return to the processing shownin FIG. 19.

On the other hand, in the event that the representative image selectionunit 103 b makes a positive decision in the step S260, the controldevice 103 causes the flow of control to proceed to a step S270, inwhich the representative image selection unit 103 b makes a decision asto whether or not only a single image with proper brightness has beenincluded among the images without blurring in which the main subject isfocused on. In the event that the representative image selection unit103 b makes a negative decision, the control device 103 causes the flowof control to proceed to the step S280 described above, in which therepresentative image selection unit 103 b specifies an image captured inthe optimal composition from among the images without blurring in whichthe main subject is focused on having proper brightness. After that, thecontrol device 103 causes the flow of control to proceed to the stepS230 described above, in which the representative image selection unit103 b selects the image captured in the optimal composition specified inthe step S280 as a representative image, and causes the flow of controlto return to the processing shown in FIG. 19.

On the other hand, in the event that the representative image selectionunit 103 b makes a positive decision in the step S270, the controldevice 103 causes the flow of control to proceed to the step S230described above, in which the representative image selection unit 103 bselects the single image specified as an image with proper brightness inthe decision processing of the step S270 as a representative image, andcauses the flow of control to return to the processing shown in FIG. 19.

Next, the processing when the representative image selection unit 103 bmakes a negative decision in the step S210 will be explained. In thiscase, the control device 103 causes the flow of control to proceed to astep S290, in which the representative image selection unit 103 b makesa decision as to whether or not an image with proper brightness has beenincluded in the target folder. In the event that the representativeimage selection unit 103 b makes a negative decision, the control device103 causes the flow of control to proceed to the step S280 describedabove, in which the representative image selection unit 103 b specifiesthe image captured in the optimal composition from among the imagesrecorded in the target folder. After that, the control device 103 causesthe flow of control to proceed to the step S230 described above, therepresentative image selection unit 103 b selects the image captured inthe optimal composition specified in the step S280 as a representativeimage, and causes the flow of control to return to the processing shownin FIG. 19.

On the other hand, in the event that the representative image selectionunit 103 b makes a positive decision in the step S290, the controldevice 103 causes the flow of control to proceed to a step S300. In thestep S300, the representative image selection unit 103 b makes adecision as to whether or not only a single image with proper brightnesshas been included in the target folder. In the event that therepresentative image selection unit 103 b makes a negative decision, thecontrol device 103 causes the flow of control to proceed to the stepS280 described above, in which the representative image selection unit103 b specifies the image captured in the optimal composition from amongthe images with proper brightness recorded in the target folder. Afterthat, the control device 103 causes the flow of control to proceed tothe step S230 described above, in which the representative imageselection unit 103 b selects the image captured in the optimalcomposition specified in the step S280 as a representative image, andcauses the flow of control to return to the processing shown in FIG. 19.

On the other hand, in the event that the representative image selectionunit 103 b makes a positive decision in the step S300, the controldevice 103 causes the flow of control to proceed to the step S230described above, in which the representative image selection unit 103 bselects the single image specified as an image with proper brightness inthe decision processing of the step S300 as a representative image, andcauses the flow of control to return to the processing shown in FIG. 19.

According to the fourth embodiment explained above, the followingoperations and advantageous effects can be achieved.

(1) It is arranged that image data having been grouped on apredetermined condition in advance are sorted into folders for each ofthe groups and recorded in the HDD 104, and the subject recognition unit103 a carries out the subject recognition processing for the imagerecorded in the HDD 104 so as to recognize the subject included in theimage. It is arranged that, based upon the recognition result by thesubject recognition unit 103 a, the representative image selection unit103 b then sets a selection criterion to select a representative imagefrom among the image files recorded in each of the folders, and basedupon the set selection criterion, the representative image selectionunit 103 b selects the representative image from among the image filesrecorded in each of the folders. As a result, based upon the selectioncriterion having been set based upon the recognition result of thesubject, an appropriate image as a representative image can be selectedfrom among the images included in the group.

(2) It is arranged that the representative image selection unit 103 bcauses the thumbnail image of a representative image to be arranged anddisplayed on the folder icon so as to display information related to therepresentative image and information related to the group that includesthe representative image in association with each other. As a result, byviewing the thumbnail image, the user can comprehend what image groupthe image recorded in the folder belongs to.

(3) It is arranged that the representative image selection unit 103 bmakes a decision as to whether or not an image in which a person iscaptured has been included in the target folder, and, when a decision ismade that an image in which a person is captured has been included,carries out the “representative image selection processing in the caseof a person photographed” whilst, when a decision is made that an imagein which a person is captured has not been included, carries out the“representative image selection processing in the case of a person notphotographed”. As a result, the selection criterion can be set withpriority given to whether a person highly likely to be the main subjecthas been captured.

(Variation 21)

It is to be noted that the image display device of the embodimentdescribed above can be modified as follows.

(1) In the embodiment described above, it is arranged that therepresentative image selection unit 103 b makes a decision as to whetheror not a person is captured in an image, and, based upon the decisionresult, carries out the “representative image selection processing inthe case of a person photographed” shown in FIG. 20 or the“representative image selection processing in the case of a person notphotographed” shown in FIG. 21. Then, it is arranged that the conditionto select the representative image (selection criterion) is changedbetween the “representative image selection processing in the case of aperson photographed” and the “representative image selection processingin the case of a person not photographed”. However, it may also bearranged that a part of the condition to select the representative imageis in common between the “representative image selection processing inthe case of a person photographed” and the “representative imageselection processing in the case of a person not photographed”.

In other words, also in the “representative image selection processingin the case of a person photographed”, it may also be arranged that atleast one of the conditions of whether the image is blurring, whetherthe main subject is focused on, whether the image has proper brightness,and whether the image is captured in the optimal composition is added soas to select the representative image. In addition, it may also bearranged that another condition is added to “representative imageselection processing in the case of a person photographed” and“representative image selection processing in the case of a person notphotographed” so as to select the representative image.

(2) In the embodiment described above, an example was explained inwhich, as shown in FIG. 18, the representative image selection unit 103b causes the thumbnail image of a representative image to be arrangedand displayed on the folder icon so as to display information related tothe folder that includes the representative image and informationrelated to the representative image in association with each other.However, it may also be arranged that the representative image selectionunit 103 b uses another method so as to display information related tothe folder that includes the representative image and informationrelated to the representative image in association with each other. Forinstance, it may also be arranged that the representative imageselection unit 103 b causes the folder name of the folder that includesthe representative image and the thumbnail image of the representativeimage to be displayed in association with each other.

(3) In the embodiment described above, an example in which a personalcomputer is used as the image display device 100 was explained. However,it may also be arranged that another device or system that can record animage file and display an image is used, for example, a digital camera,a mobile phone, a photo storage device, an online album service usingthe Internet, or the like.

Fifth Embodiment

In the fourth embodiment, an example was explained, in which, for afolder in which an image file has already been stored, a representativeimage is selected from among the images stored (recorded) in the saidfolder. In the fifth embodiment, a selection method of a representativeimage will be explained, in which a new image file is stored (added) inthe folder in which the representative image has already been selected.The representative image selection unit 103 b in the fifth embodimentselects the representative image in the following order (1) to (3).

(1) The representative image selection unit 103 b saves in the HDD 104the information obtained when it selected the representative image thathas already been selected (information obtained when carried out theprocessing shown in FIG. 19 to FIG. 21) associated with the said folder.The information to be saved includes the following.

[A] The case in which an image in which a person is captured exists inthe said folder (a positive decision is made in the step S20 (FIG. 19))

The information to be saved includes information showing the presence orabsence of an image in which a family member or an acquaintance iscaptured, the number of the said image, the presence or absence of animage in which a face is facing the front, the number of the said image,the presence or absence of an image of a smile, the number of the saidimage, and the area of a face in an image in which the area of the faceis the greatest.

[B] The case in which an image in which a person is captured does notexist in the said folder (a negative decision is made in the step S20(FIG. 19))

The information to be saved includes information showing the presence orabsence of an image determined not to be blurring, the number of thesaid image, the presence or absence of an image in which the mainsubject is focused on, the number of the said image, the presence orabsence of an image with proper brightness, the number of the saidimage, the presence or absence of an image captured in the optimalcomposition, and the number of the said image.

(2) The representative image selection unit 103 b performs the decisionprocessing (the same manner as the step S20) as to whether or not aperson is captured in an image file to be newly stored (added) in thesaid folder.

(3) In the event that a person is captured in the image to be newlystored (added), the representative image selection unit 103 b carriesout the processing shown in FIG. 20. At this time, the representativeimage selection unit 103 b selects the image representing the saidfolder using information whether the family member or the acquaintancehas been captured in the image to be newly stored (added), whether theface captured in the image to be newly stored (added) is facing thefront, whether the face captured in the image to be newly stored (added)is a smile, the area of the face captured in the image to be newlystored (added), and the information saved in the HDD 104.

In the event that a person is not captured in the image to be newlystored (added), the representative image selection unit 103 b carriesout the processing shown in FIG. 21. At this time, the representativeimage selection unit 103 b selects the image representing the saidfolder using information showing whether the image to be newly stored(added) is an image without blurring, whether the main subject of theimage to be newly stored (added) is focused, whether brightness of theimage to be newly stored (added) is proper, the image to be newly stored(added) is in the optimal composition, and the information saved in theHDD 104.

According to the fifth embodiment explained above, an appropriaterepresentative image can be selected when a new image file is stored(added) in the folder in which the representative image has beenselected. Use of the information saved in the HDD 104 thus allows theprocessing load of the representative image selection unit 103 b and theprocessing time to be reduced compared to the case in which each of thedecision processing is re-performed without using the saved information.

It is to be noted that it may be arranged that, when an image in which aperson is not captured is newly stored (added) in the folder in which animage in which a person is captured is stored, the previousrepresentative image in the said folder is designated to be arepresentative image as it is without carrying out the processing shownin FIG. 20. On the other hand, when an image in which a person iscaptured is newly stored (added) in the folder in which an image inwhich a person is not captured is stored, the processing shown in FIG.20 is carried out, so that the image selected through the processingshown in FIG. 20 (i.e., the new image in which the person is captured)is designated as a representative image, in place of the previousrepresentative image in the said folder.

Sixth Embodiment

In the sixth embodiment, the case will be explained in which arepresentative image is selected for the folder in which an image forwhich image correction processing such as exposure compensation andimage blur reduction is carried out is stored (recorded). Therepresentative image selection unit 103 b in the sixth embodimentselects the representative image in the following order (1) to (2).

(1) The representative image selection unit 103 b carries out theprocessing shown in FIG. 19 to FIG. 21 for the folder in which therepresentative image is to be selected, more specifically, therepresentative image selection unit 103 b performs the subjectrecognition processing (step S10) and the decision processing (step S20)whether a person is captured for all the images recorded in the targetfolder, and, in the event that there is an image in which a person iscaptured, carries out the processing shown in FIG. 20, whilst in theevent that there is not an image in which a person is captured, carriesout the processing shown in FIG. 21.

(2) When the processing shown in FIG. 20 has been carried out, therepresentative image selection unit 103 b causes the representativeimage selected in the processing of FIG. 20 to be arranged and displayedin the folder icon (step S60). The representative image selection unit103 b causes the representative image selected in the processing of FIG.21 to be arranged and displayed in the folder icon (step S60).

According to the sixth embodiment described above, selection of arepresentative image can be performed appropriately even for the folderin which an image for which image processing such as exposurecompensation and image blur reduction is carried out is stored.

(Variation 22)

It may also be arranged that the following processing (A) to (G) isperformed in place of the processing of the step S280 in the sixthembodiment (to specify an image captured in the optimal composition fromamong images recorded in as folder) or in combination with theprocessing of the step S280. In the event that a plurality of images forwhich image processing such as exposure compensation and image blurreduction is carried out are stored, a plurality of images with properbrightness and without image blurring exist, and then it becomesdifficult to specify a single image. In variation 22, therefore, any ofthe following processing (A) to (G) is performed so as to make an imageeasy to specify. It is to be noted that a plurality of the processing(A) to (G) may be arbitrarily combined.

(A) The representative image selection unit 103 b specifies an imagecaptured during a period of time in which shooting frequency is greater.For instance, with reference to shooting time information recorded asadditional information, e.g., Exif information in the image file, therepresentative image selection unit 103 b calculates the distribution ofshooting time of an image recorded in the said folder so as to specifythe image captured during a period of time in which the shooting timesare most concentrated. Since a great shooting frequency indicates that asubject of high concern or interest to the user has been highly likelyto be captured, such an image is preferable as a representative image.

(B) The representative image selection unit 103 b specifies an imagehaving a greater number of pixels. For example, the representative imageselection unit 103 b calculates pixel number information (data size) ofan image recorded in the said folder so as to specify the finest image.A clear image is preferable as a representative image.

(C) The representative image selection unit 103 b specifies the imagethat has been viewed more frequently. For instance, with reference toviewing frequency information for each image recorded in the saidfolder, the representative image selection unit 103 b specifies theimage that has been viewed most frequently. In this case, therepresentative image selection unit 103 b counts the number of times inwhich the image file is read out (the number of times of access) afterthe image file was recorded in the said folder, and causes countinginformation for each image file to be associated with the said folderand to be saved in the HDD 104. Since a frequent access indicates that asubject of high concern or interest to the user has been highly likelyto be captured, such an image is preferable as a representative image.

(D) The representative image selection unit 103 b specifies an imagewhose edge histogram or color histogram is nearer to the average valueof all the images recorded in the said folder. Since information of theaverage histogram of the images in the folder is highly likely toreflect the characteristics of a subject image of high concern orinterest to the user, an average image is preferable as a representativeimage.

(E) The representative image selection unit 103 b calculates thedistribution of shooting time of images recorded in the said folder soas to specify the image with the earliest shooting time.

(F) The representative image selection unit 103 b calculates thedistribution of shooting time of images recorded in the said folder soas to specify the image with the last shooting time.

(G) The representative image selection unit 103 b calculates thedistribution of shooting time of images recorded in the said folder soas to specify the image captured at the middle of a predetermined periodof time among those included in the said period of time.

Seventh Embodiment

In the seventh embodiment, the case will be explained in which an imagestored in the folder for which the representative image has already beenselected undergoes image correction processing such as exposurecompensation and image blur reduction after the representative image hasbeen selected. In this case, although the representative image selectionunit 103 b newly performs selection processing of a representative imageafter the image correction processing has been performed, it carries outonly the processing shown in FIG. 20 or the processing shown in FIG. 21without carrying out the processing shown in FIG. 19.

More specifically, the processing shown in FIG. 20 is carried out if animage in which a decision is made that a person is captured has beenselected as the previous representative image, and the processing shownin FIG. 21 is carried out if an image in which a decision is made that aperson is not captured has been selected as the previous representativeimage.

According to the seventh embodiment described above, the representativeimage can be selected again according to requirements in the event thatthe image processing such as exposure compensation and image blurreduction is carried out afterwards for the image in the folder forwhich the representative image has already been selected.

In the seventh embodiment, it may also be arranged that, when an imagein which a decision is made that a person is not captured has beenselected as the previous representative image, the previousrepresentative image in the said folder is designated as arepresentative image as it is without newly performing the selectionprocessing of the representative image (in this example, the processingshown in FIG. 20). This is because, when at least only one of the imageblur reduction and the exposure compensation has been performed as imagecorrection processing, even if the processing shown in FIG. 20 isperforated for the folder that includes the image having undergone thesaid image correction processing, the representative image to beselected is the same as the representative image of the said folderselected previously (prior to the image correction processing).

However, in the event that processing (for instance, smile correctionprocessing or the like) other than the image blur reduction exposurecompensation has been performed as image correction processing, theselection processing of the representative image (in this example, theprocessing shown in FIG. 20) is newly performed. This is because, whenthe processing shown in FIG. 20 is performed for the folder thatincludes the image having undergone the said smile correctionprocessing, the representative image to be selected may be differentfrom the representative image of the said folder selected previously(prior to the smile correction processing).

Although a variety of embodiments and variations have been explained inthe above explanations, the present invention is not limited to those.Other modes which may be conceived of within the range of the technicalconcept of the present invention are also included within the range ofthe present invention. In addition, the embodiments and the variationsdescribed above may be arranged arbitrarily in combination therewith.

The disclosures of the following priority applications are hereinincorporated by reference:

-   Japanese Patent Application No. 2008-8990 (filed Jan. 18, 2008)-   Japanese Patent Application No. 2008-113706 (filed Apr. 24, 2008)-   Japanese Patent Application No. 2008-113707 (filed Apr. 24, 2008)-   Japanese Patent Application No. 2008-113708 (filed Apr. 24, 2008)-   Japanese Patent Application No. 2008-113709 (filed Apr. 24, 2008)

1-9. (canceled)
 10. An image display device, comprising: a subjectrecognition unit configured to carry out subject recognition processingfor image data, which are grouped, classified and recorded in eachgroup, so as to recognize a subject included in an image; a setting unitconfigured to set a selection criterion to select a representative imagefrom image data recorded in each group based upon a recognition resultby the subject recognition unit; and a selection unit configured toselect the representative image from image data recorded in each groupbased upon the selection criterion set by the setting unit.
 11. An imagedisplay device according to claim 10, further comprising: a display unitconfigured to display information related to the representative imageselected by the selection unit and information related to a group thatincludes the representative image in association with each other.
 12. Animage display device according to claim 11, wherein: the setting unit isconfigured to make a decision as to whether or not an image in which aperson is captured is included in each group based upon the recognitionresult by the subject recognition unit; when a decision is made that animage in which a person is captured is included, the setting unit sets afirst selection criterion as the selection criterion; and when adecision is made that an image in which a person is captured is notincluded, the setting unit sets a second selection criterion as theselection criterion.
 13. An image display device according to claim 12,wherein: the first selection criterion is a selection criterion toselect the representative image based upon a face of a person capturedin an image.
 14. An image display device according to claim 13, wherein:the first selection criterion is a selection criterion to select therepresentative image by making a decision as to whether or not a face ofthe person is a face of a family member or an acquaintance, whether ornot a face of the person is facing frontward, whether or not a face ofthe person is smiling, and whether or not an area of a face of theperson is of a quality that is better than a predetermined level ofquality.
 15. An image display device according to claim 12, wherein: thesecond selection criterion is a selection criterion to select therepresentative image by making a decision as to whether or not the imageis not blurring, whether or not the image has a main subject in focus,whether or not the image is with proper brightness, and whether or notthe image is captured in a composition better than a predeterminedquality level of composition.
 16. An image display device according toclaim 10, wherein the subject recognition unit, the setting unit and theselection unit are implemented in a processor.
 17. A non-transitorycomputer readable medium having an image display program embeddedtherein, the program, when executed by a computer, causing the computerto carry out: a subject recognition procedure that carries out subjectrecognition processing for image data which are grouped, classified andrecorded in each group so as to recognize a subject included in animage; a setting procedure that sets a selection criterion to select arepresentative image from image data recorded in each group based upon arecognition result by the subject recognition procedure; and a selectionprocedure that selects the representative image from image data recordedin each group based upon the selection criterion set by the settingprocedure.
 18. A non-transitory computer readable medium having an imagedisplay program according to claim 17, further comprising: a displayprocedure that displays information related to the representative imageselected by the selection procedure and information related to a groupthat includes the representative image in association with each other.19. A non-transitory computer readable medium having an image displayprogram according to claim 18, wherein: the setting procedure makes adecision as to whether or not an image in which a person is captured isincluded in each group, based upon a result of a subject recognition bythe subject recognition procedure; when a decision is made that an imagein which a person is captured is included, the setting procedure sets afirst selection criterion as the selection criterion; and when adecision is made that an image in which a person is captured is notincluded, the setting procedure sets a second selection criterion as theselection criterion.
 20. A non-transitory computer readable mediumhaving an image display program according to claim 19, wherein: thefirst selection criterion is a selection criterion to select therepresentative image based upon a face of a person captured in an image.21. A non-transitory computer readable medium having an image displayprogram according to claim 20, wherein: the first selection criterion isa selection criterion to select the representative image by making adecision as to whether or not a face of the person is a face of a familymember or an acquaintance, whether or not a face of the person is facingfrontward, whether or not a face of the person is smiling, and whetheror not an area of a face of the person is of a quality that is betterthan a predetermined level of quality.
 22. A non-transitory computerreadable medium having an image display program according to claim 19,wherein: the second selection criterion is a selection criterion toselect the representative image by making a decision as to whether ornot the image is not blurring, whether or not the image has a mainsubject in focus, whether or not the image is with proper brightness,and whether or not the image is captured in a composition better than apredetermined quality level of composition.